Assessing the Quality of Model Differencing Engines

In recent years many tools and algorithms for model comparison and differencing were proposed. Typically, the main focus of the research laid on being able to compute the difference in the first place. Only very few papers addressed the quality of the delivered differences sufficiently. Hence, this is a general shortcoming in the state-of-the-art. Currently, there are no established community standards how to assess the quality of differences and it is neither possible to compare the quality of different algorithms, nor can developers decide whether or not an algorithm is able to produce adequate results in a given application scenario. We propose a parallel working session to be held to discuss this general problem and its implications. The goal of the working session is to achieve a common understanding of what the crucial factors in assessing the quality of differences are. Furthermore, it is planed to discuss possible solutions that help the research community as whole, e.g. by drafting the design of an initial benchmark corpus which later could be turned into a standardized, openly available benchmark set.